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NEW  METHODOLOGIES  PROGRAM; 


FINAL  REPORT 

< • 

1 September  1975  to  31  October  1977 

I.  INTRODUCTION 


This  is  a Final  Report  for  Contract  No.  F44620-76-C-0008, 
"Studies  of  New  Methodologies  for  Applied  Human  Factors 
Research"  with  Canyon  Research  Group,  Inc.,  for  the  period 
1 September  1975  to  31  October  1977. 

Dr.  Charles  W.  Simon  was  the  Principal  Investigator. 

Dr.  Alfred  R.  Fregly  was  Contract  Monitor  for  AFOSR. 


II.  PROGRAM  OBJECTIVES 

The  objectives  to  be  achieved  during  this  two-year  period 
were:  1)  devise  strategies  for  employing  "advanced  experi- 

mental methodologies"  developed  on  this  and  earlier  AFOSR 
supported  programs*;  2)  upgrade  earlier  reports  on  the  basis 
of  the  knowledge  acquired  since  their  completion;  3)  bring 
together  the  techniques  and  strategies  into  a cohesive  document. 


*These  techniques  from  earlier  programs  were  summarized 
in  C.  W.  Simon,  Final  Report,  Advanced  Methodologies  Study 
Program,  Contract  No.  F44620-72-C,  Culver  City,  CA;  Hughes 
Aircraft  Company,  1 July  1972  to  30  June  1975. 


III.  SUMMARY  OF  RESULTS 


New  Reports 

During  the  period,  the  following  reports  were  written  and 
published; 

Simon,  C.  W.  Response  Surface  Methodology  Revisited; 

A Commentary  on  Research  Strategy.  Westlake  Village, 

CA;  Canyon  Research  Group,  Inc.,  Technical  Report 
No.  CWS-01-76,  July  1976.  (60  pages) 

Simon,  C.  W.  Analysis  of  Human  Factors  Engineering 
Experiments;  Characteristics,  Results  and  Applica- 
tions. Westlake  Village,  CA;  Canyon  Research  Group, 

Inc.,  Technical  Report  CWS-02-76,  August  1976.  (104  pages) 

Simon,  C.  W.  Design,  Analysis,  and  Interpretation  of 
Screening  Studies  for  Human  Factors  Engineering  Research. 
Westlake  Village,  CA;  Canyon  Research  Group,  Inc.,  Tech- 
nical Report  No.  CWS-03-77,  September  1977.  (220  pages) 

Simon,  C.  W.  New  Research  Paradigm  for  Applied  Experi- 
mental Psychology;  A System  Approach.  Westlake  Village, 
CA;  Canyon  Research  Group,  Inc.,  Technical  Report  No. 
CWS-04-77,  October  1977.  (123  pages) 

These  reports  will  be  described  in  the  Program  Review 
section  below. 

Revised  Reports 

During  this  period,  two  reports  written  on  earlier  Advanced 
Methodologies  programs  for  AFOSR  were  revised: 
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Simon,  C.  W.  Economical  Multifactor  Designs  for  Human 
Factors  Engineering  Research.  Culver  City,  CA;  Hughes 
Aircraft  Company,  TR  # P73-326,  June  1973. 

Simon,  C.  W.  Methods  for  Handling  Sequence  Effects  in  ' 

Human  Factors  Engineering  Research,  Culver  City  CA:  Hughes  i 

Aircraft  Company,  TR  # P74-541,  December  1974.  | 

i 

Changes  were  made  on  2 3 and  21  pages,  resp 'ctively , and  the  | 

report  numbers  were  changed  by  adding  the  letter  "A"  to  the 
original  number  of  each.  Copies  of  the  revised  report  were 
sent  to  document  reference  and  distribution  centers,  including 
DDC,  ERIC,  NASA-STIF,  NTIS,  and  the  Psychological  Abstracts. 

Seminars 

Three  more  seminars  on  "Advanced  Experimental  Methodologies" 
were  given  during  this  period  to  the  following  organizations: 

Human  Factors  Laboratory,  Naval  Training  Equipment 
Center,  Orlando,  Florida.  15,  16,  17  June  1976. 

Studies  and  Analysis  Department,  Hughes  Aircraft 
Company,  Fullerton,  California.  6,  7,  8 July  1976. 

Display  Systems  Department,  Hughes  Aircraft  Company, 

Culver  City,  California.  October  1976. 

Each  of  these  organizations  subsequently  employed  techniques 
learned  from  the  "Advanced  Experimental  Methodologies"  seminar 
in  their  research  proposals  and/or  experiments. 
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IV . PROGRAM  REVIEW 


The  four  reports  prepared  for  this  program  each  make  a 
unique  contribution  to  the  objectives  cited  in  Section  II. 

The  content  of  these  reports,  when  properly  interpreted  and 
used,  should  result  in  a marked  improvement  in  the  quality 
of  information  obtained  from  formal  psychology  experiments. 

A summary  of  each  report  is  given  below. 

1.  Analysis  of  Human  Factors  Engineering  Experiments  (CWS-02-76) . 

Two  hundred  thirty-nine  experiments  published  in  the 
journal  Human  Factors  during  the  period  from  1958  to  1972 
were  analyzed  for  the  purpose  of  discovering  the  characteris- 
tics of  their  experimental  plans,  the  quality  and  character  of 
their  results,  and  the  degree  to  which  these  results  had  been 
applied  to  real  systems.  The  analysis  revealed  that  these 
experiments  investigated  too  small  an  experimental  space, 
showed  essentially  no  diversity  in  their  selection  of  a basic 
experimental  design,  collected  far  more  redundant  data  than  was 
needed,  and  failed  to  properly  handle  the  irrelevant  variance 
arising  from  sequence  effects. 

IVhen  the  experimental  results  were  reanalyzed  to  discover 
what  proportion  of  the  observed  performance  variance  was 
accounted  for  by  experimental  factors,  on  average,  the  pro- 
portion was  much  less  than  half.  This  proportion  increased, 
on  average,  as  more  factors  were  studied  in  an  experiment 
although,  for  any  size  experiment,  there  were  always  some 
experiments  in  which  the  experimental  factors  accounted  for 
practically  none  of  the  observed  variance  and ‘ some  in  which 
they  accounted  for  most  of  it.  There  was  empirical  evidence 
from  these  experiments  to  show  that  third-order  or  higher 
interactions  had  only  negligible  effects  on  performance;  this 
was  so  even  though  many  of  these  sources  had  been  found  to  be 
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"statistically  significant."  Nearly  a quarter  of  the  main 
effects  accounting  for  less  than  one  percent  of  the  total 
variance  were  statistically  significant.  Subjects  used  to 
replicate  an  experiment  (and  thus  presumed  to-be  homogeneous)* 
generally  accounted  for  much  more  of  the  performance  variance 
than  specifically  selected  factors  of  subject  characteristics. 

When  a survey  was  made  of  those  who  conducted  the  experi- 
ments, it  was  discovered  that  slightly  more  than  half  of  the 
experiments  had  been  done  to  find  answers  of  general  applica- 
bility. Less  than  a third  of  the  experiments  were  known  or 
believed  to  have  influenced  the  design  of  a real  system.  A 
majority  of  the  investigators  said  they  would  not  do  their 
experiments  any  differently  if  they  were  to  repeat  them  today. 

Numerical  data  in  support  of  these  and  other  results  are 
supplied  along  with  some  limited  discussion  on  the  implications 
of  this  analysis  for  an  improved  experimental  methodology. 

This  represents,  it  is  believed,  the  first  extensive  collection 
of  "hard"  data  on  these  topics. 


2.  Response  Surface  Methodology  Revisited.  (CWS-01-76) 

This  report  critically  evaluates  a series  of  five  published 
papers  that  purported  to  explain  and  illustrate  the  character- 
istics and  applications  of  central-composite  designs  (CCDs)  in 
the  context  of  response-surface  methodology  (RSM) . The  series 
was  important  because  it  succeeded  in  arousing  among  human 
factors  investigators  considerable  interest  in  this  powerful 
experimental  methodology.  A detailed  evaluation  revealed, 
however,  that  the  experimental  papers  in  that  ‘series  1)  fail 
to  illustrate  the  most  important  and  useful  features  of 
"response  surface  methodology"  designs  as  proposed  by  G.E.P. 

Box  and  his  associates;  2)  employ  questionable  procedures  not 


specific  to  RSM  that  permit  interpretations  of  results  not 
considered  by  the  investigators;  and  3)  do  not  constitute  an 
experimental  evaluation  of  the  effectiveness  of  RSM  central- 
composite  designs  as  suggested  by  the  investigators.  - Because* 
the  published  experiments  were  being  used  as  models  of  good 
research  employing  this  methodology,  which  they  were  not,  this 
report  was  written  to  alert  potential  users  to  their  weaknesses, 
as  they  affected  both  application  and  interpretation,  and  to 
offer  constructive  guidance.  The  distinction  between  using  an 
experimental  design,  that  is,  a pattern  of  data-collection 
points,  and  employing  an  experimental  strategy  is  emphasized. 

3.  Design  and  Analysis  of  Screening  Studies.  (CWS-03-77) 

In  this  report,  methods  are  described  for  constructing 
Resolution  IV  screening  designs  that  are  robust  to  linear, 
quadratic,  and  cubic  trend  effects  and  will  also  keep  the 
number  of  factor-level  changes  for  some  variables  at  a minimum. 
Complete  designs,  capable  of  screening  up  to  32  variables,  are 
provided  along  with  appropriate  methods  cf  analysis.  Numerous 
criteria  for  selecting  non-trivial  variables  are  discussed, 
including:  mean  difference,  eta  squared,  cumulative  probability, 
and  half-normal  plots.  How  to  handle  subjects  in  an  experiment 
is  described  when  their  characteristics  can  be  included  as 
experimental  factors  and  when  they  cannot  be,  and  when  sub- 
jects are  merely  a-  form  of  replication.  The  values  of 
introducing  multiple  center  points  into  the  2 fractional 
factorial  screening  designs  are  discussed,  particularly  as  they 
are  used  to  build  the  screening  design  into  a central-composite 
design.  Lack-of-fit  tests  are  provided  to  help  decide  whether 
a second-  or  a third-order  response  surface  is  needed.  Numer- 
ous methods  of  analyzing  screening  studies  with  multiple 
responses  are  described.  A method  is  given  for  developing  a 
prediction  equation  with  data  collected  from  an  incompleted 
screening  design. 
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4.  New  Research  Paradigm  for  Experimental  Psychology.  (CWS-04-77) 


To  spotlight  the  crisis  occurring  in  scientific  psychology, 
warnings,  complaints,  and  criticisms  by  prominent  psychologists 
and  non-psychologists  are  quoted.  Traditional  experimental 
method  of  the  past  hundred  years  does  not  produce  the  data 
needed  to  solve  problems  faced  by  a modern  society.  The  results 
from  many  experiments  cannot  be  combined  into  a modular,  quan- 
titative data  base. 

The  traditional  experimental  method,  as  it  is  applied  in 
engineering  psychology,  is  critically  examined.  Characteristic 
procedures  and  concepts  are  analyzed  to  show  why  they  have  been 
counterproductive  to  the  ultimate  experimental  goals  — 
prediction  of  field  performance  and  aggregation  of  data  for 
future  use. 

The  two  distinct  approaches  to  data  collection  used  by 
empirical  psychologists,  "Experimental"  and  "Correlational," 
are  defined  and  compared.  Arguments  are  presented  for  merging 
the  more  productive  features  of  each.  However,  contrary  to  I 

suggestions  in  the  past,  the  combined  approach  would  emphasize  | 

the  point  of  view  of  the  Experimentalists.  This  means  that 
the  manipulative  method  would  be  retained  but  in  a way  that 
would  enable  the  holistic  philosophy  of  the  Correlationists 
to  dominate. 

Philosophy,  strategy,  and  techniques  are  described  that 
can  be  combined  into  a new  research  paradigm  for  experimental  j 

psychology.  A sequential  process  is  proposed  that  enables 
systematic  multifactor  experiments  to  be  performed  with  great 
economy.  Problems  are  first  defined  with  a real-world  orien- 
tation. Next,  using  primarily  manipulative  procedures,  fifty 
to  one-hundrqd  candidate  factors  from  equipment,  environment, 
personnel,  and  task  sources,  can  be  screened  systematically  to 
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identify  the  non-trivial  ones.  These  non-trivial  factors  for 
the  particular  task  are  then  subjected  to  further  investigation, 
the  data  from  which  being  combined  with  that  from  the  screening 
study  to  produce  a response  surface  as*  defined  by  a polynomial 
of  the  appropriate  degree.  This  equation  is  then  refined,  min- 
imizing both  bias  and  random  error,  the  fiducial  limits 
determined,  and  the  resulting  product  verified  under  operational 
conditions.  Further  refinement  may  be  required. 

The  feature  that  makes  this  paradigm  unique  is  that  the 
total  data-collection  process  for  deriving  an  equation  of  all 
critical  variables  affecting  an  operational  task  will  ordin- 
arily be  less  than  that  used  in  four  and  five  factor  experiments 
using  traditional  methodology.  The  conseuqences  are  that 
prediction  from  laboratory  data  to  field  performance  becomes  a 
reality  and  a quantitative  data  base  for  future  reference  can 
be  constructed. 


V.  CONCLUSIONS 

Not  only  were  the  objectives  of  the  immediate  program 
met  during  this  period,  but  also  some  of  the  long-range  objec- 
tives set  seven  years  ago  at  the  beginning  of  the  Advanced 
Methodologies  Program,  and  those  of  the  Principal  Investigator 
set  twenty-five  years  ago.  It  is  now  possible  to  systematically 
study  the  effects  of  twenty-five,  fifty,  or  even  one-hundred 
variables  deemed  potentially  critical  to  a particular  task,  and 
to  do  so  with  considerable  economy  in  the  data-collection 
effort  while  meeting  all  relevant  scientific  standards.  No 
practical  technique  has  ever  been  available  before  that  would 
permit  the  experimental  psychologist  to  investigate  problems 
of  such  complexity  in  such  a systematic  and  economical  manner. 
The  paradigm  for  this  is  described  in  one  of  the  reports 
completed  this  year. 
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Two  conditions,  however,  mar  the  conclusiveness  of  this 
success.  One,  the  paradigm  is  uneven  in  the  detail  in  which 
it  has  been  worked  out.  This  must  be  corrected  to  facilitate 
its  use.  Until  that  time,  an  investigator  who  wants  to 
apply  the  method  must  be  prepared  to  smooth  these  gaps  himself 
Two,  efforts  to  date  to  apply  the  paradigm  have  shown  evidence 
of  faulty  designs,  incomplete  analyses,  and  erroneous  inter- 
pretations. These  have  not  been  due  to  any  inadequacy  in  the 
paradigm  or  methodologies  involved,  but  instead  have  reflected 
inadequate  preparation  on  the  part  of  investigators  before 
they  begin  their  experiments.  In  spite  of  all  this,  the 
response  to  the  total  program  has  been  positive  enough,  par- 
ticularly by  the  small  group  who  have  read  the  reports  and 
attended  the  seminars,  to  conclude  with  great  confidence  that 
this  new  approach  to  experimental  psychology  will  survive  and 
flourish. 
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